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(57) This invention relates to improvements in 
so-called "virtual reality" computer systems. 
The display system described enables the effec- 
tive integration of computer generated images 
and real, naturally occurring images in the visu- 
al display units which provide the user with his 
entire visual stimulation. Areas of the user's 
notional field of view where com- 
puter-generated imagery is required (for 
example the cockpit canopy in an aircraft flight 
simulator) are coloured with a predetermined 
colour or intensity. Two video cameras are pro- 
vided, each of which is mounted so that its field 
of view corresponds to a respective one of the 
notional fields of view of the user's eyes. The 
signals from the video cameras are analysed to 
determine the presence of the predetermined 
colour or intensity, thereby giving an indication 
that a computer generated image is required for 
that part of the video camera's field of view. If 
the predetermined colour or intensity is not 
present, the signal from the video camera is 
relayed directly to the appropriate one of the 
visual display units. 



Fig. 2 
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This invention relates to improvements in the per- 
ception of computer generated imagery, and more 
particularly to the effective integration of computer 
generated images with real, naturally occurring im- 
ages in helmet mounted visual display units which 
provide the user with his entire visual stimulation. 

Recent developments in computer technology 
have enabled so-called "virtual reality" computer sys- 
tems to be produced. These systems provide com- 
plete visual stimulation for the user usually by means 
of a helmet mounted display apparatus comprising a 
pair of miniature Visual Display Units (VDUs) ar- 
ranged to project their images into respective ones of 
the user's eyes such that the image displayed is per- 
ceived stereoscopicaliy thereby creating the illusion 
that the image is three-dimensional. These systems 
allow the user to interact with the displayed image by 
providing a means of monitoring movement of the 
user's head and, in some examples, movement of his 
eyes and other parts of his body. The information ob- 
tained by monitoring these movements allows the 
system to generate and present different images to 
the respective eyes of the user corresponding to 
changes in his line of sight. An example of such a sys- 
tem is given in British patent no. 2201069. 

Current virtual reality systems present the user 
with a graphics display which is entirely computer 
generated. The computer graphics system generates 
the complete visual image for each of the VDUs in the 
helmet mounted display. In many applications highly 
complex images need to be generated which, com- 
bined with the necessity to have them produced in 
real time, makes very heavy demands on computing 
power in terms of memory size and speed of access. 
One particularly notable problem is that of realistical- 
ly representing the user's hands, legs and feet when- 
ever the user's current line of sight would include 
those parts of his body. The result is that the systems 
are extremely expensive or they provide simpler im- 
ages than are desirable for optimum realism. 

Attempts have been made to overcome the need 
for high computing power by laying the computer gen- 
erated image over the real image of a scene by using 
an optical combiner to combine the two images. Al- 
though these systems are less demanding in terms of 
what is required of the computer generated images 
because the images may then occupy a smaller area, 
the system now must determine which part of the 
scene viewed by the userthe computer generated im- 
age should occupy as his line of sight changes as a 
result of eye and head movements. Current head pos- 
ition sensors are limited in terms of positional accu- 
racy, and, when used with such visually coupled sys- 
tems, any errors in the calculation of head position 
cause the computer generated image to move rela- 
tively and unrealistically within the real scene. Under 
certain circumstances this image motion can be ex- 
tremely objectionable and may even cause nausea. 



An object of the present invention is to provide an 
improved system whereby a real scene can be over- 
laid with computer generated images. 

According to the present invention there is pro- 

5 vided a display system for presenting composite im- 
ages consisting of a real portion and a computer-gen- 
erated portion to a user's eyes, the system compris- 
ing visual display unit means for presenting the com- 
posite images; sensor means for producing a direc- 

10 tion signal indicative of the direction of the user's field 
of view; video camera means for producing a se- 
quence of video signals which in combination are rep- 
resentative of a field of view thereof, said video cam- 
era means being mounted such that the field of view 

15 of t he video camera means is substantially coincident 
with and follows movement of the field of view of the 
user; image generating means for producing the com- 
puter-generated portion of the composite image in 
accordance with said direction signal; computer 

20 means for analysing each of said video signals to de- 
termine by comparison with stored reference data 
whether each video signal is indicative of an area of 
the field of view of the video camera means which the 
computer-generated image is to occupy and for pro- 

25 ducing corresponding switching signals; and image 
combining means responsive to said switching sig- 
nals for producing the composite image, the real part 
of which is derived from said video signals. 

Preferably said video camera means is mounted 

30 on the user's head for movement therewith. 

Alternatively said video camera is mounted for 
movement in response to said direction signal. 

Preferably, said visual display unit means com- 
prises two visual display units which present different 

35 composite images to respective ones of the user's 
eyes; said video camera means comprises two video 
cameras each of which have a field of view which is 
substantially coincident with the fields of view of re- 
spective ones of the user's eyes; said image gener- 

40 ating means produces two computer-generated por- 
tions, said computer means analyses the video sig- 
nals from respective ones of said two video cameras; 
and said image combining means produces two dif- 
ferent composite images which, when presented to 

45 respective ones of the user's eyes, are perceived 
stereoscopicaliy. 

Optionally, the stored reference data relates to a 
predetermined colour and said computer means ana- 
lyses each of said video signals by determining the 

so colour thereof and comparing the colour with the pre- 
determined colour to produce a switching signal 
whenever the colours match. 

Alternatively the stored reference data relates to 
a predetermined intensity and said computer means 

55 analyses each of said video signals by determining 
the intensity thereof and comparing the intensity with 
the predetermined intensity to produce a switching 
signal wheneverthe intensities match or one exceeds 
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the other according to a predetermined rule. 

Advantageously said visual display unit means 
provides substantially complete visual stimulation for 
the user. 

For a better understanding of the invention, an 5 
embodiment of it will now be described by way of ex- 
ample only, and with particular reference to the ac- 
companying drawings, in which:- 

Figure 1 shows a schematic diagram of a display 
system according to the present invention; 10 
Figure 2 shows an aircraft cockpit interior with 
which the display system of Figure 1 is used; and 
Figure 3 shows a more detailed schematic dia- 
gram of the computer generated and real image 
combining circuit shown in Figure 1. 15 
To improve understanding of the drawings, like 
elements which appear in more than one figure are 
designated by the same reference number. 

Figure 1 shows schematically a display system 
for combining computer generated and real images 20 
on two miniature VDUs (not shown) in a user's helmet 
mounted display 1. The system includes a miniature 
video camera 3 which is connected to camera control 
unit 4 for passing images of a viewed scene, such as 
an aircraft cockpit interior shown in Figure 2, thereto. 25 
The camera 3 is mounted on the helmet mounted dis- 
play 1 so that its field of view is altered when the user 
moves his head. The camera control unit 4 contains 
all the electronics which does not have to be included 
in the body of the camera 3 (to save weight) and con- 30 
verts the composite video signal received from cam- 
era 3 into three primary (red, green and blue) colour 
signals 6. These signals are applied to a computer 
generated and real image combining circuit 7, some- 
times referred to as a "chroma key" system. Also ap- 35 
plied to the combining circuit 7 are three primary col- 
our signals 8 received from image generating comput- 
er 9 which receives at a control input signals repre- 
senting the position of the user's head (not shown) 
calculated by head position determining circuit 11. A 40 
sensor 1 3 mounted on helmet mounted display 1 , and 
which therefore moves with the user's head, provides 
signals representing head movement to the position 
determining circuit 11. An example of a suitable sen- 
sor is sold by Polhemus Navigation Sciences, a divi- 45 
sion of McDonnel Douglas Electronics Company, un- 
der the name 3space (Trade Mark) Isotrack. The sig- 
nals 6 and 8 are combined in the combining circuit 7 
to provide a combined output 1 5 which comprises the 
real scene viewed by camera 3 overlaid with the com- 50 
puter generated image. 

If the system is to provide different images to 
each of the VDUs in the helmet mounted display 1 
such that the image is perceived stereoscopically by 
the user, an additional video camera, camera control 55 
unit, image combining circuit and possibly image gen- 
erating computer are required. These additional ele- 
ments are configured as those described above. For 



the sake of brevity the operation to produce just one 
combined image will be described. This single image 
could be displayed on both the VDUs in the helmet 
mounted display 1 but obviously it is desirable to pro- 
vide the user with a stereoscopic image. 

Figure 2 shows part of an aircraft cockpit interior 
1 7 which is used as part of a flight simulation system. 
The user 19 of the system sits in the normal flying 
position in seat 21 . With the helmet mounted display 
removed, the user's 19f ield of view would include the 
view through the cockpit canopy 23, the instrument 
panel 25 and the joystick 27. However, when the dis- 
play system is in use the user 19 wears the helmet 
mounted display 1 and the two miniature VDUs con- 
tained therein provide him with his total visual stimu- 
lation. 

Clearly, in order to provide flight simulation, real- 
istic computer generated images must be displayed in 
the area occupied by the cockpit canopy 23; whereas 
the remainder of what would be in the pilot's field of 
view could be viewed naturally without affecting real- 
ism. In order that the display system can identify 
areas where a computer generated image is required, 
such areas are coloured in a single colour which does 
not appear elsewhere in the field of view. In this ex- 
ample the cockpit canopy 3 is coloured blue and uni- 
formly illuminated, although any colour which meets 
the above criterion could also be used. 

Camera 3 is mounted on the helmet mounted dis- 
play 1, and is therefore moved with the user's 19 
head such that the camera 3 has a field of view which 
is equivalent to that of one of the user's 19 eyes. (Ob- 
viously for stereoscopic vision two cameras would be 
provided, one mounted on either side of the helmet 
mounted display 1 and each having a field of view 
equivalent to that of respective ones of the user's 
eyes.) The signals from the camera 3 are converted 
into the three colour signals 6 by camera control unit 
4. The signals 6 are then passed to combining circuit 
7. Also passed to the combining circuit 7 are the three 
colour signals 8 which make up the computer gener- 
ated image produced by image generating computer 
9. This image generating computer 9 is of a known de- 
sign and is programmed to produce an image which 
occupies the area of the cockpit canopy using signals 
received from the head position determining circuit 
11. These signals allow the computer 9 to calculate 
the required composition, size and shape of the com- 
puter generated image. 

Figure 3 shows the contents of the computer gen- 
erated and real image combining circuit 7 enclosed by 
dotted lines. The operation is as follows. The signal 6 
derived from the field of view of camera 3 is passed 
to detector circuit 29. The operation of circuit 29 is 
such that it generates an output signal 31 whenever 
a specified colour occurs in signals 6. This colour is 
selected by a control 33 which sets the appropriate 
percentages of red, green and blue so that it is the 
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same colour as that of cockpit canopy 23. The output 
signal 31 , the strength of which is proportional to the 
closeness of t he colour to the colour set by control 33, 
is passed to the positive input of comparator 35. The 
negative input of comparator 35 is connected to 
threshold control 37 which, by setting the voltage at 
the negative input, establishes what strength of sig- 
nal 31 is to cause the comparator 35 to switch from 
zero voltage output (indicating that the specified col- 
our is not present) to its saturation output (indicating 
that the specified colour is present). The comparator 
35 output controls the operation of switch 37 which 
selects either the image from the video camera 3 
(represented by signals 6), or from the image gener- 
ating computer 9 (represented by signals 8), at a giv- 
en instant on the raster scan of the video camera 3 
for output to conventional video amplifier 39 which 
produces a drive standard 75 ohm output video signal 
which is fed to helmet mounted display 1. The signal 
8 from the image generating computer 9 is also proc- 
essed by circuit 4 1 to produce a pulse output 43 which 
is fed to camera control unit 4 in order to synchronise 
its raster scan with the video signals produced by the 
image generating computer 9. 

The effect, therefore, of the combining circuit 7 is 
to continuously monitor the output signal 6 derived 
from the video camera 3 for the presence of the col- 
our chosen for the cockpit canopy 23, the presence 
of this colour being indicative that a computer gener- 
ated image is required for that position, and, in re- 
sponse to that monitoring, selecting for output at that 
point on the raster scan to the user's 19 helmet 
mounted display 1 either the signal from the video 
camera 3 or from the image generating computer 9. 

One of the advantages of this embodiment over 
the known optical combiner referred to above, is that 
the video camera provides the user with the real (i.e. 
not computer generated) part of the image. This en- 
ables the colour of the video signal to be analysed so 
that the computer generated image never occupies 
an area which should form a "rear part of the image 
irrespective of any errors in the sensor 13. 

The computer generated and real image combin- 
ing circuit could be adapted to work with composite 
video inputs (PAL), but this would require additional 
circuitry to extract the primary colour information and 
to compensate for phase delays in this circuitry. 

As an alternative to mounting the video camera 
3 on the user's helmet mounted display 1, the camera 
could instead be mounted on a fixed part of cockpit 
interior 17 by way of an electrically controlled servo- 
motor driven mounting 45 (shown ghosted in Figure 
1). The signal (shown by the dotted line in Figure 1) 
from the position determining circuit 11 is then also 
applied to the mounting 45 to cause the camera 3 to 
move such that it has a field of view which is equiva- 
lent to the user's eyes. The movement could be pro- 
vided by way of gimbals, for example. One of the ad- 
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vantages of mounting the camera 3 on the cockpit in- 
terior is that the movement of the video camera 3 and 
the input of image generating computer 9 are both 
driven by signals derived from the same helmet 

5 mounted sensor 1 3 and thus any errors in the meas- 
urement of the position of the user's head affect both 
the video camera 3 and the image generated comput- 
er 9 in the same way. Hence, the problem of the com- 
puter generated image moving relatively within the 

10 real scene, which is associated with conventional 
computer generated and real image combining sys- 
tems, may be overcome. 

In addition to detecting head movement, the sen- 
sor 13 may include a further sensor contained in the 

15 helmet mounted display 1 which detects movement of 
the user's eyes. This additional information would be 
fed to the image generating computer 9 via position 
determining circuit 11 in a similar manner to the head 
position determining information. In the case where 

20 the camera 3 is mounted on the helmet mounted dis- 
play 1 , it will need to be movable in response to an eye 
position signal from the position determining circuit 
11. In the case where the camera 3 is mounted on a 
fixed part of the cockpit interior, the eye position sig- 

25 nal will simply comprise an additional element to the 
signal from the position determining circuit 11 which 
is fed to the mounting 45. 

It will be understood that the aircraft cockpit of 
the flight simulator can be changed to allow the user 

30 to gain experience of a range of aircraft without the 
need to alter the display system because the areas 
where a computer image is required are simply des- 
ignated by their blue colouring. 

Further, the instrument panel displays could be 

35 coloured so that a computer generated image ap- 
pears thereon in order that they give a display indi- 
cative of aircraft flight. 

In cases where the ergonomic design of a pro- 
posed cockpit interior requires testing before manu- 

40 facture, t he cockpit interior may itself be generated by 
the image generating computer, leaving the pilot's 
body as the only "real" part of the image. In this way 
the legibility of displays from different angles of view 
could be determined. 

45 If a "data glove" such as that described in US pa- 

tent number 4,988,981 (Zimmerman et al) is worn on 
the user's hand the accessibility of various controls 
could also be tested. A data glove includes sensors for 
detecting the gestures of the hand as well as a hand 

so position sensor positioned on the glove. The signals 
from these sensors are fed to the image generating 
computer 9 so that the position of the user's hand and 
any gestures made can be used to effect the comput- 
er generated image - for example by showing the op- 

55 eration of a switch on a control panel. Unlike the sys- 
tem described in the aforesaid US patent number 
4,988,981, the embodiment described obviates the 
. need to produce a computer generated representa- 
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tion of the user's hand. 

As an alternative to detecting a particular colour 
to determine the correct position of computer gener- 
ated imagery, the intensity of light could instead be 
measured and used to determine positioning. In this 
way, the cockpit canopy 23 could be made of a highly 
radiation absorbent material or highly illuminated to 
designate that computer generated imagery is re- 
quired in this area. 

The system described could obviously be used in 
many other applications. For example, the system 
could be used inside a motor vehicle with the image 
generating computer 9 producing the views from the 
vehicle's windows. In this case the windows would be 
coloured and uniformly illuminated in a selected col- 
our, thereby allowing the computer generated and 
real image combining circuit 7 to distinguish between 
areas where computer generated images and real im- 
ages are required. 



Claims 

1. A display system for presenting composite im- 
ages consisting of a real portion and a computer- 
generated portion to a user's eyes, the system 
comprising visual display unit means for present- 
ing the composite images; sensor means for pro- 
ducing a direction signal indicative of the direc- 
tion of the user's field of view; video camera 
means for producing a sequence of video signals 
which in combination are representative of a field 
of view thereof, said video camera means being 
mounted such that the field of view of the video 
camera means is substantially coincident with 
and follows movement of the field of view of the 
user; image generating means for producing the 
computer-generated portion of the composite im- 
age in accordance with said direction signal; 
computer means for analysing each of said video 
signals to determine by comparison with stored 
reference data whether each video signal is indi- 
cative of an area of the field of view of the video 
camera means which the computer-generated 
image is to occupy and for producing correspond- 
ing switching signals; and image combining 
means responsive to said switching signals for 
producing the composite image, the real part of 
which is derived from said video signals. 

2. A display system according to claim 1, wherein 
said video camera means is mounted on the 
user's head for movement therewith. 
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A display system according to claim 1, 2 or 3 
wherein the stored reference data relates to a 
predetermined colour and said computer means 
analyses each of said video signals by determin- 
ing the colour thereof and comparing the colour 
with the predetermined colour to produce a 
switching signal whenever the colours match. 

A display system according to any preceding 
claim, wherein said visual display unit means 
comprises two visual display units which present 
different composite images to respective ones of 
the user's eyes; said video camera means com- 
prises two video cameras each of which have a 
field of view which is substantially coincident with 
the fields of view of respective ones of the user's 
eyes; said image generating means produces 
two computer-generated portions, said computer 
means analyses the video signals from respec- 
tive ones of said two video cameras; and said im- 
age combining means produces two different 
composite images which, when presented to re- 
spective ones of the user's eyes, are perceived 
stereoscopically. 

A display system according to any one of claims 
1 to 3, wherein the stored reference data relates 
to a predetermined intensity and said computer 
means analyses each of said video signals by de- 
termining the intensity thereof and comparing 
the intensity with the predetermined intensity to 
produce a switching signal whenever the intensi- 
ties match or one exceeds the other according to 
a predetermined rule. 

A display system according to any preceding 
claim, wherein said visual display unit means pro- 
vides substantially complete visual stimulation 
for the user. 

A display system substantially as hereinbefore 
described with reference to and as illustrated in 
the accompanying drawings. 



A display system according to claim 1, wherein 
said video camera is mounted for movement in re- 
sponse to said direction signal. 
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